Definition

Consider RV F=f1,f2,...,fKF={f_1,f_2,...,f_K}, with probability pk=Prob{F=fK}p_k=\mathrm{Prob}\{F= f_K\}

Self-Information of one realization fk:Hk=−log⁡(pk)f_k : H_k= -\log(p_k)

  • pk=1p_k=1: always happen, no information
  • Pk∼0P_k \sim 0: seldom happen, its realization carries a lot of information

Entropy = average information

H(ℱ)=−∑f∈𝒜pℱ(f)log2⁡pℱ(f)H(\mathcal{F})=-\sum_{f \in \mathcal{A}} p_{\mathcal{F}}(f) \log_2 p_{\mathcal{F}}(f)
  • Entropy is a measure of uncertainty or information content, unit=bits
  • Very uncertain -> high information content

Notes

References

  1. https://en.wikipedia.org/wiki/Entropy_(information_theory)
  2. https://ee.stanford.edu/~gray/it.pdf